Add TypeScript coordinator runtime#69079
Conversation
503a0e1 to
cb139b3
Compare
|
@uranusjr @jason810496 PR for TS sdk runtime. |
The runtime should keep generated supervisor payloads as the source of truth while making the discriminators required where the coordinator constructs or narrows wire messages.
cb139b3 to
5de8276
Compare
|
Need to fix the static check failure obviously but otherwise very nice work |
Thank you. I have been spending the last couple of months understanding the Task SDK execution path, edge worker work, and coordinator model, so it’s really encouraging to see this coming together. I’ve pushed a follow-up for the static check failure. |
Yes, the multi-lang dag will work after this PR. Just the bundler and packer needs to be implemented so you will have to create airflow-metadata.yml file. The readme in the example describes how to run it |
jason810496
left a comment
There was a problem hiding this comment.
Nice! Here're all the follow-up issues that I saw during the review so far. I prefer to have real issues to keep track of them instead of having follow-up statements in the review comments that might be missed in the long run.
- #69286 - TypeScript SDK: add a
ts packbundle build tool - #69287 - TypeScript SDK: Dag/task-aware multi-bundle selection in NodeCoordinator
- #69288 - TypeScript SDK: native TypeScript Dag declaration (remove Python stub requirement)
- #69289 - TypeScript SDK: coordinator schema-mapping tests (camelCase to snake_case)
- #69290 - TypeScript SDK: Airflow-level integration tests for coordinator-mode tasks
- #69291 - TypeScript SDK: refactor shared logic between NodeCoordinator and ExecutableCoordinator
- #69292 - TypeScript SDK: log socket buffering and reconnect hardening
|
(Note that native dag declaration should wait on AIP-82 Dag Importers) |
Yes, I just added at the top of the issue. |
Adds the TypeScript coordinator runtime to
@apache-airflow/ts-sdk, thefollow-up to #67908 (TypeScript task SDK public interface). This lets Airflow
execute registered TypeScript task handlers through a bundled Node.js entrypoint.
(length-prefixed msgpack frames) over the
--command--logssockets:connects, receives the task startup message, dispatches to the registered
handler for the Dag/task pair, and reports the terminal task state.
XCom (get / set, including the automatic
return_valuepush for non-undefinedhandler returns), and Connections.
ctx.signal(AbortSignal); SIGTERM/SIGINTabort the signal with a grace period before force-exit.
src/generated/supervisor.ts) plus agenerate:supervisorscript, and a./coordinatorpackage export subpath.example/showing a Python stub Dag + TypeScript handlers, bundledwith esbuild, and documents the bundle layout and Airflow coordinator config.
This is intentionally still Python-stub-Dag mode: TypeScript Dag declaration is
not introduced here. TypeScript registers handlers for Dag/task IDs declared by
Python stub tasks. The example bundles with esbuild and writes
airflow-metadata.yamlmanually; folding metadata generation into the buildscript is left to a separate follow-up.
Testing performed:
Next steps